Method and system for generating training data

ABSTRACT

A method for generating training data can include: determining a set of images; determining a set of masks based on the images; determining a first mesh based on the set of masks; optionally determining a refined mesh by recomputing the first mesh; optionally determining one or more faces of the refined mesh; optionally adding one or more keypoints to the refined mesh; optionally determining a material property set for the object; optionally generating a full object mesh; determining one or more scenes; optionally determining training data based on the one or more scenes; optionally training one or more object detectors using the training data; and detecting one or more objects using the trained object detector.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.63/051,816, filed 14 Jul. 2020, US Provisional Application No.63/051,867, filed 14 Jul. 2020, US Provisional Application No.63/051,872, filed on 14 Jul. 2020, and US Provisional Application No.63/143,414, filed 29 Jan. 2021, each of which is incorporated in itsentirety by this reference.

TECHNICAL FIELD

This invention relates generally to the computer vision field, and morespecifically to a new and useful method for generating training data.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a schematic representation of a variant of the method.

FIG. 2 is a schematic representation of a variant of the system.

FIG. 3 is a schematic representation of a variant of the system.

FIG. 4 depicts an illustrative example of an embodiment of the method.

FIG. 5 depicts an embodiment of the method.

FIG. 6 depicts an example of the system.

FIG. 7 depicts an example of the system.

FIGS. 8A and 8B depict examples of object keypoints.

FIG. 9 depicts an example of S400 and S700.

FIGS. 10A and 10B depict examples of S700.

FIG. 11 depicts an embodiment of the method.

FIG. 12 depicts an embodiment of the method.

FIG. 13 depicts an embodiment of S240.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The following description of the preferred embodiments of the inventionis not intended to limit the invention to these preferred embodiments,but rather to enable any person skilled in the art to make and use thisinvention.

1. Overview

As shown in FIG. 1, the method for training data generation can include:determining a set of images S100; generating an object mesh using theset of images S200; optionally determining one or more object componentsof the refined mesh S300; optionally adding one or more keypoints to theobject mesh S400; optionally determining a material property set for theobject S500; optionally modifying the object mesh Shoo; determiningtraining data S700; optionally training one or more object detectorsusing the training data S800; detecting one or more objects using thetrained object detector S900; and/or any other suitable elements.

The method functions to generate synthetic training data to identifyindividual instances of previously-unknown objects in unstructuredscenes. This synthetic training data can optionally be used to train oneor more object detectors to identify robotic grasping points, identifygraspable faces, and/or select a grasping point on one or moreindividual object instances.

2. Examples

In a first example, the method for training data generation can include:determining a set of images for an object, wherein the set of imagesincludes RGB images and depth images; generating an object mesh usingthe set of images; determining object components (e.g., object face,object part, etc.) based on a bounding box and/or local features of theobject; adding one or more keypoints to each object component of theobject mesh and/or tagging mesh faces, wherein each keypoint isassociated with an object face identifier for the respective objectcomponent; determining a material property set for the object, whereinthe material property set includes scores for surface properties (e.g.,shininess, transparency, color, texture, etc.); optionally modifying theobject mesh based on the material property set and/or the set of images;determining one or more virtual scenes using the object mesh, whereinthe object mesh can be the full object mesh (e.g., 360 degree view ofthe object) or include multiple meshes for different sides of the object(e.g., wherein determining virtual scenes can include tracking gravityvectors to make sure that known surfaces are visible to virtualcameras); determining training data (e.g., rendering synthetic imagesthat depict object instances using the set of images and the materialproperty set, wherein each object instance is associated with unoccludedobject keypoints) based on the one or more virtual scenes; training oneor more object detectors for object grasping using the training data;using the trained object detectors to detect the object keypoints in aphysical scene (e.g., detect nonoccluded object keypoints, nonoccludedobject faces, etc.); and grasping objects at the keypoints using therobotic arm and end effector.

In a first specific example (e.g., as depicted in FIG. 12), the imagescan be determined using an image capture environment that includes: ahousing defining an enclosed measurement volume; a rotating pedestalmounted to the housing within the measurement volume and configured tosupport an object in the measurement volume; one or more calibrationfeatures, wherein a calibration feature can be mounted to each side (orsubset thereof) of the pedestal; an imaging system configured to capturethe set of images; an imaging system actuator mounting the imagingsystem and configured to move the camera along a predefined trajectoryrelative to the support; and a plurality of light emitting elementsmounted to the housing, wherein the light emitting elements are evenlydistributed about the interior of the housing.

In a second specific example, generating the object mesh using the setof images can include: determining a set of masks based on the images,wherein each mask includes the object segmented from the rest of theimage; determining a refined mesh based on the set of masks and thedepth images using visual hull carving and/or depth fusion; furtherrefining the refined mesh using one or more smoothing techniques, andafter smoothing, further refining the mesh using the set of masks;optionally generating a full object mesh by aligning multiple refinedmeshes of the same object.

3. Benefits

Variants of the system and method can confer several benefits overconventional systems.

First, variants of the system and method can provide training dataincluding unstructured piles of objects (e.g., the same object,different objects, etc.). The training data can include a syntheticimage and a training label (or training target), wherein the traininglabel can include unoccluded object keypoints for each unoccludedsection of an object that is visible (e.g., that is in a virtual camerafield of view) in the synthetic image (e.g., as depicted in FIGS. 10Aand 10B).

Second, variants of the system and method can provide accurate trainingdata (e.g., with accurate object segments, keypoints, and labels) fortraining one or more object detectors. This accurate training data canbe generated using geometrically accurate object meshes, which can alsobe generated by the system and method. In variants, these geometricallyaccurate object meshes (and/or accurate geometric models) can begenerated without complex geometric scans or CAD models, and can begenerated on-premises for unknown objects (e.g., using only 2D imagesand optionally depth measurements). The accuracy of the object mesh caninfluence the accuracy of the one or more object detectors to identifyan object in a pile and successfully execute an object grasp, sohigh-accuracy meshes can be desirable in some applications. In variants,accurate object meshes can be automatically generated by both extractingconvex and concave object surfaces from images and depth maps by using acombination of visual hull carving, depth fusion algorithms. In thesevariants, accurate masks, generated by using a combination of depthintegration and image segmentation techniques, can be used to cropimages and/or depth maps used by the visual hull carving and depthfusion algorithms to increase the accuracy of the resultant object mesh.Variants of the system and method can optionally determine accuratemeshes for transparent and/or reflective objects, which areconventionally difficult to model from images.

Third, variants of the system and method can add keypoints and/or objectcomponent identifiers to object meshes, such that when the training datais generated, objects depicted in the training data can be labelled withthe keypoints and optionally, the object component identifiers. Theobject component identifiers can be individual object faces and/orcomponents (e.g., cap, body, arm, head, leg, etc.). The object detectorscan be trained to identify said keypoints, which can be used todetermine a grasp for the object during inference.

Fourth, variants of the technology can provide a scalable system and/ormethod that can automatically generate the object meshes, identifygraspable faces, generate training data, and train object recognitionnetworks without manual input (e.g., without humans in the loop). Thisallows for users without any machine learning expertise to easily addnew objects to an object recognition repository and/or reconfigure apacking line to identify new objects.

However, variants of the system and method can confer any other suitablebenefits.

4. System

The method is preferably performed using the system 20, as shown in FIG.2, including: an image capture environment 100, one or more imagingsystems 200, a computing system 300, an end effector 400, a robotic arm500, and/or any other suitable components. However, the method can beperformed by any other suitable system.

The image capture environment can define a controlled environment forimage capture of objects. The image capture environment can include ameasurement volume, a housing, one or more light emitting elements, oneor more imaging systems, one or more imaging system actuators, apedestal, one or more calibration features, and/or any other suitablecomponents.

The image capture environment preferably includes a housing, but canadditionally or alternatively not include a housing (e.g., be an openspace). The housing can be: a spherical segment (e.g., hemisphere),cylindrical (e.g., with an axis perpendicular gravity), prismatic,and/or have other geometry. The housing can be defined by a set of walls(e.g., top, base, lateral sides, back, etc.) surrounding the measurementvolume, or not be defined by walls (e.g., be an open space). The wallsare preferably radiused (curved, such as concave or convex toward themeasurement volume). However, the walls can be wavy, planar and meet atperpendicular junctions (corners), or have any other suitableconfiguration. The walls can be made of metal, plastic glass, ceramic,or any other suitable material or combination thereof. The walls canhave an anti-reflective coating (e.g., matte finish), be reflective, orhave other material properties. The walls can be white, black, or haveany other suitable color. The walls can have one or more openings,wherein the opening can accommodate placement of an object within thehousing. The opening can be half of a wall, a third of a wall, a quarterof a wall, and/or any other suitable fraction of one or more walls. Theopening can be on a top wall, lateral side walls, a back wall, and/orany other suitable wall. However, the housing can be otherwise defined.

The measurement volume (e.g., which can be defined by the housing) canfunction to provide a volume to sample images of an object (e.g., asdepicted in FIG. 6). The measurement volume can be back lit, front lit,fully lit (e.g., from at least two or more opposing and/or adjacentsides), and/or otherwise illuminated by the light emitting elements. Themeasurement volume can define a coordinate system. The coordinate systemcan be relative to: a middle of the object support (e.g., pedestal), thecenter of the measurement volume, a camera (e.g., wherein the coordinatesystem moves with the camera), and/or any other suitable referencepoint. In a first variant, the measurement volume can be an open volume.In a second variant, the measurement volume can be an enclosed volume,defined by the housing. However, the measurement volume can be otherwiseconfigured.

The pedestal can function to support the object in the measurementvolume (e.g., as depicted in FIG. 7). The pedestal preferably actuatesrelative to the camera, the measurement volume, and/or any othersuitable system component, but can additionally or alternatively bestatic. The pedestal preferably rotates (e.g., such that the camera cancapture a 360 degree view of the object), but can alternatively linearlytranslate (e.g., along the x, y, and/or z axes), or otherwise actuate.The pedestal can include one or more surfaces (e.g., a top plane, bottomplane, front plane, back plane, side plane(s), etc.). The pedestal canhave any suitable geometry, such as the cross section of the pedestalcan be a: triangle, square, rectangle, pentagon, hexagon, heptagon,octagon, 9-sided polygon, decagon, and/or any other suitable shape. Eachside or a subset thereof of the pedestal (e.g., top, vertical side,etc.) can include one or more calibration features, or not include acalibration feature. The pedestal preferably includes a top plane,wherein the object can be placed on the top plane for image capture. Thepedestal can be mounted to the bottom, top, side, and/or other portionof the housing or measurement volume. In variants, the pedestal can be:a column, an arm, and/or any other suitable support. The pedestal canoptionally include a weight sensor (e.g., load cell, etc.) configured tomeasure the weight or mass of the object. However, the pedestal can beotherwise configured.

The calibration feature can function to enable camera calibration, imagealignment, object pose identification, and/or any other functionality.The calibration feature can be positioned on the one or more surfaces ofthe pedestal, on a surface that supports the pedestal (e.g., as depictedin FIG. 7), on the housing, and/or in any other suitable location in theimage capture environment. The calibration feature can include: arucomarkers, chessboard, charuco markers, halcon pattern, and/or any othersuitable fiducial or calibration pattern. However, the calibrationfeature can be otherwise configured.

The one or more light emitting elements can function to illuminate(e.g., front-light, back-light, etc.) the measurement volume and/orobject during image capture. The light emitting elements can be pointlights, diffused lights, and/or any other suitable light type. The lightemitting elements can emit light having constant color temperature,saturation, color, or any other suitable lighting parameter. Thelighting parameters can be constant or be adjustable (e.g., by theprocessing system, etc.). The lights can emit white light (e.g., coolwhite light, warm white light, etc.), light in the visible spectrum, IRlight, UV light, multispectral light, hyperspectral light, or lighthaving any other suitable wavelength. The light emitting elements canbe: light emitting diodes (e.g., gallium arsenide, gallium arsenidephosphide, aluminum gallium arsenide phosphide, gallium phosphide,aluminum gallium phosphide, etc.), filament lamps, discharge lights,fluorescent lights, and/or any other suitable light emitting element.The light emitting elements can be evenly distributed, unevenlydistributed, and/or otherwise distributed about the measurement volume.The light emitting elements can be arranged in a pattern: a bar, an arc,hemisphere, randomly, ring (e.g., about an imaging system), and/orarranged in any other suitable pattern.

The one or more light emitting elements can be arranged in light sets(e.g., a pattern, line, shape, etc.). The light sets can be indexed andcontrollable, or be controlled as a unitary population. A light set caninclude one or more lights. Light sets can overlap (e.g., share lights)or not overlap. Light sets can be arranged: on the pedestal (e.g., top,sides, within the pedestal, etc.), on the housing interior, on anactuatable bar (e.g., imaging system actuator or separate bar), aboutthe imaging system, and/or otherwise arranged. The light sets can bearranged such that the lights emit light from the behind or beneath theobject and/or pedestal relative to the imaging system, such that theobject and/or pedestal occludes light. The light sets can include afront set, a back set, a top set, a bottom set, left set, right set,and/or any other suitable set (e.g., relative to the measurement volume,relative to the pedestal, relative to the camera, etc.). The light setscan be arranged equidistant to the pedestal, different distances fromthe pedestal, or otherwise arranged relative to the pedestal (e.g.,equidistant, different distances, etc.). The light emitting elements canbe mounted: to the housing (e.g., interior thereof), the measurementvolume, the imaging system support or actuator, the pedestal, and/or toany other suitable component. The light sets can be mounted to the top,bottom, left, right, back, front, interior, and/or exterior of themounting component. In a first variant, the light emitting elements canbe mounted to the housing interior (e.g., entire interior surface,arcuate strip, point lights, etc.). In a second variant, the lightemitting elements can be mounted to a bar (e.g., arcuate bar) that movesrelative to the pedestal. In a third variant, the light emittingelements can be mounted to the pedestal. However, the light emittingelements can be otherwise configured.

The imaging system can function to capture images of the object. Theimages can be: RGB, multispectral, hyperspectral, images in apredetermined set of wavelengths, greyscale, depth, and/or any othersuitable image. The imaging system can include one or more cameras, oneor more depth sensors, and/or any other suitable imaging sensors. Theimaging system can be statically mounted (e.g., to an imaging systemactuator, to the housing, etc.) relative to the measurement volume, beactuatable to the measurement volume, or otherwise mounted relative tothe measurement volume. The imaging system is preferably mounted to animaging system actuator, but can additionally or alternatively bemounted to the interior of the housing, to an imaging system actuator,to the pedestal, and/or to any other suitable surface.

The camera can be: a stereo camera pair, CCD, CMOS, RGB-D, and/or anyother suitable camera. The one or more cameras can be evenly or unevenlydistributed within the measurement volume. When the camera actuates, theimaging system actuator can be the same as the bar that the lights aremounted to, be a robotic arm, and/or any other suitable actuator. In afirst variant, the one or more cameras can be mounted to the housing. Ina second variant, the one or more cameras can be mounted to an imagingsystem actuator that moves relative to the housing, relative to thepedestal, and/or relative to any other system component (e.g., examplesare depicted in FIGS. 5 and 6). In a specific example, the camera(s) canbe mounted to an arcuate arm that sweeps between the front and the back(or between the left and right) of the pedestal and/or object. However,the camera can be otherwise configured.

The depth sensor can function to determine depth measurements (e.g.,depth maps) associated with the images captured by the camera. The depthmap can be pixel-aligned with the RGB images, or be misaligned. Thedepth sensor can be an IR camera, ToF system, structured light system,projected light system, and/or any other depth sensor. The depth sensorcan be calibrated to create a mapping between the depth measurement andan image pixel, an image captured by the camera, and/or otherwisecalibrated. The depth sensor can be collocated with the camera, notcollocated with the camera, or alternatively, the depth sensor can bethe camera (e.g., wherein the camera is an RGB-D camera). However, thedepth sensor can be otherwise configured.

The imaging system is preferably mounted to an imaging system actuatorthat functions to move the camera within the measurement volume relativeto the pedestal. The imaging system actuator can actuate the imagingsystem to capture a plurality of views of the object, wherein theplurality of views cooperatively form a photo dome or photosphere of theobject (e.g., inverse photosphere, inverse photosphere, photodome orphotosphere depicting more than a threshold proportion or angular regionof the exterior of the object, etc.). The imaging system actuator canbe: an arcuate bar, beam, boom, and/or any other suitable structure. Theimaging system actuator can include a motor and/or any other suitablecomponents. The imaging system actuator can include one or more jointsor not include joints. The imaging system actuator can be mounted to theinterior of the housing, mounted to the pedestal (e.g., side of thepedestal), and/or otherwise mounted to the image capture environment.The imaging system actuator can be: plastic, metal, wood, and/or anyother suitable material. However, the imaging system actuator can beotherwise configured.

In a first example, the system includes a housing defining an arcuateenclosed measurement volume, a rotary pedestal arranged within themeasurement volume and configured to receive an object thereon, anoptical system (e.g., including a color camera and/or a depth sensor)mounted to an actuatable arm (e.g., a rotatable arm), and a set oflights arranged along the arcuate surface of the measurement volumeinterior (e.g., evenly distributed; arranged along the edges or centerof the measurement volume arc). The measurement volume material can be:textured (e.g., diffuse or scatter incident light, create diffusereflections), matte, mirrored (e.g., create specular reflections),function as optical waveguides, and/or have other optical properties.The measurement volume can be white, black, and/or any other suitablecolor.

The computing system can function to perform the method described below.The computing system can be a local computing system, a remote computingsystem (e.g., network connected servers), and/or any other suitablecomputing system.

The computing system can optionally function to execute or include oneor more processing modules. Processing modules can include: maskgeneration module (e.g., configured to perform S200), a mesh generationmodule (e.g., configured to perform S200), a physics module, a renderingmodule (e.g., configured to perform S700), and/or any other suitablemodule. The computing system can additionally include a control module,which can function to move the pedestal, imaging system(s), robotic arm,end effector, and/or any other system components. The control module canfunction to control the light emitting elements (e.g., fully light theobject, front light the object, back light the object, and/or otherwiselight the object during image capture). The control module can turn onor off the light emitting elements, capture images, and/or perform anyother suitable functionality. The control module is preferablyelectrically connected to the pedestal, imaging systems, light emittingelements, and/or any other component. However, the control module can beotherwise configured. However, the computing system can be otherwiseconfigured.

The mask generation module can function to generate a mask of the object(e.g., in each image, for the image set, etc.). The mask generationmodule can perform S200, but can additionally or alternatively performother processes. The mask generation module can leverage one or more of:neural networks (e.g., CNNs, RNNs, etc.), machine learning algorithms,and/or other mask generation methods.

The mesh generation module can function to generate a meshrepresentative of the object's geometry. The mesh generation module canperform S200, but can additionally or alternatively perform otherprocesses. The mesh generation module can leverage one or more of:neural networks (e.g., CNNs, RNNs, etc.), machine learning algorithms,structure-from-motion algorithms, stereo methods, shape-from-shadingmethods, photogrammetry, active methods, passive methods, and/or othermethods.

The physics module can function to simulate a scene using one or morerefined meshes (e.g., configured to perform S900). The physics modulecan be used with a bullet simulation, PhysX simulation, BeamNGsimulation, and/or any other suitable simulation. However, the physicsmodule can be otherwise configured.

The rendering module can function to render the scene simulated by thephysics module. The rendering module can leverage one or more of: raytracing, image reprojection, mesh projection, and/or other renderingmethods.

The end effector can functions to manipulate an object. Manipulation caninclude: grasping, pushing, scooping, pulling, suctioning, dropping,and/or any other suitable manipulation. The end effector can beimpactive, ingressive, astrictive, contigutive, and/or any othersuitable type of end effector. In a first example, the end effector is asuction gripper. In a second example, the end effector is a claw gripper(e.g., dual prong, tri-prong, etc.). However, any other suitable type ofend effector can be used. The end effector can be actuated: electrically(e.g., servo/motor actuation), pneumatically, hydraulically, unactuated(e.g., passive deformation based on motion of robotic arm, rigid body,etc.), and/or otherwise actuated. However, the system can include anyother suitable end effector. The end effector is preferably mounted tothe robotic arm, but can additionally or alternatively be mounted toand/or transformed by any suitable actuation mechanism(s) (e.g., CNCgantry system, etc.) and/or in any suitable actuation axes (e.g., 6-axisrobotic actuation). However, the end effector can be otherwiseconfigured.

The robotic arm can functions to position and/or articulate the endeffector for grasping an object, but can additionally or alternativelyprovide any other suitable functionality. The robotic arm can bearticulated by automatic control and/or can be configured toautomatically execute control instructions (e.g., control instructionsdetermined based on the grasp point, dynamically determined control,etc.), however the system can alternatively be otherwise suitablycontrolled and/or otherwise suitably enable end effector articulation.The robotic arm can include any suitable number of joints which enablearticulation of the end effector in a single degree of freedom (DOF).The arm preferably includes 6 joints (e.g., a 6-axis robotic arm), butcan additionally or alternatively include seven joints, more than sevenjoints, and/or any other suitable number of joints. However, the roboticarm can be otherwise configured.

The system can be used with one or more physical objects, one or moremeshes (e.g., collection of vertices, edges, and faces that define ashape of a polyhedral object), one or more masks (e.g., 2D or 3D bitwiserepresentation such as values are 0 or 1), and/or any other components.

The physical object can be rigid, deformable, and/or include any othercharacteristic. Examples of objects that can be used include: plushtoys, plastic toys, cosmetics, and/or any other suitable object. Theobject can be unknown (e.g., no prior information is available to thesystem). Alternatively, all (or some) of the object parameters are known(e.g., material properties, such as transparency, reflectivity, etc.;whether the object has a transparent or reflective surface; size;geometry; name; etc.). The object can be associated with: an objectidentifier (e.g., assigned by a user, automatically assigned, etc.), anobject mesh (e.g., generated by the method, retrieved from a database,etc.), a set of object images and/or object image segments (e.g.,generated by S100), and/or other information.

However, the system 20 can additionally or alternatively include anyother suitable components and/or be otherwise configured.

5. Method

The method for training data generation can include: determining a setof images S100; generating an object mesh using the set of images S200;optionally determining one or more object components of the object meshS300; optionally adding one or more keypoints to the refined mesh S400;optionally determining a material property set for the object S500;optionally modifying the object mesh Shoo; determining training dataS700; optionally training one or more object detectors using thetraining data S800; detecting objects using the trained object detectorS900; and/or any other suitable elements.

The method is preferably performed by the system disclosed above, butcan be otherwise performed. The method is preferably independentlyperformed for each object, but can alternatively be performed formultiple objects at once. S100-S260 is preferably performed at leasttwice for each object: once for the object top and once for the objectbottom; however, S100-S260 can alternatively be performed once (e.g.,when the pedestal footprint is smaller than the object, when thepedestal is transparent, etc.).

5.1 Determining a Set of Images S100.

Determining a set of images S100 can function to determine one or moreimages for object mesh generation (e.g., object model reconstruction)and/or object instance rendering for generating training data. All orportions of S100 can be repeated one or more times until a terminationevent is met (e.g., the sampling sequence is completed, a predeterminednumber of object views are captured, a threshold number of images arecaptured, capture period ends, etc.). S100 is preferably performed for asingle object at a time, but can alternatively be concurrently performedfor multiple objects.

The set of images is preferably captured using the image captureenvironment, but can additionally or alternatively be captured using adifferent image capture environment, received from a separate system,retrieved from a database (e.g., wherein the images can be captured orreceived and stored in the datastore), and/or otherwise obtained. Whenthe set of images is captured using the image capture environment,determining the set of images can include calibrating the camera(s) ofthe environment before image capture, or not calibrating the cameras.

The set of images preferably cooperatively encompass all (or a subsetthereof) visible regions of the object (e.g., include images from allviewpoints of the object), such that the images cooperatively form aphoto dome, photo sphere, inverse photospehere, inverse panorama, objectpanorama, or dome of the object, but can additionally or alternativelyencompass a subset of the object viewpoints (e.g., include a top-downview, a 360 degree segment of the object sides, etc.). The viewpointsfrom which the images are captured can include: 5 viewpoints, 10viewpoints, 15 viewpoints, and/or any other suitable number ofviewpoints. Image capture at a particular viewpoint can include one ormore image captures (e.g., to capture a front lit image, a back litimage, and/or both a front lit image and back lit image). The set ofimages can include a plurality of images per side of an object (e.g., 1,10, 50, 100, 200, 300, 400, etc.), a single image per object side,and/or any other number of images. However, the set of images can beotherwise configured.

Each image preferably depicts an object, a calibration pattern, apedestal, and/or any other suitable components. Each image can becaptured by a camera of the system or of a separate system. Each imagecan be associated with a camera pose (e.g., relative to the pedestal,the measurement volume, and/or any other reference frame), pedestalpose, and/or other pose, which can be used to convert data from onereference frame to another. The pose(s) can be refined using a bundleadjustment based on the calibration pattern and/or otherwise refined.Each image can be front lit, back lit, and/or otherwise lit. Each imagecan be: RGBD, RGB, black and white, depth images, and/or any othersuitable image.

The depth images can include depths for a plurality of points in themeasurement volume, optionally surface normals for points in themeasurement volume, and/or any other information. The depth images arepreferably dense (e.g., includes a depth or height or distance to cameraper object surface depicted in each pixel), but can alternatively besparse (e.g., includes a depth for less than a threshold proportion ofthe pixels depicting the object). However, the depth images can beotherwise defined.

Each image can optionally be associated with the sampling parameters(e.g., relative lighting direction, lighting intensity, etc.) of thesystem during image capture. Sampling parameters can include: lightingparameters, pedestal parameters, camera parameters, and/or parametersfor other components. Examples of sampling parameters can include: whichlight sets to operate, lighting direction, light intensity, light color,camera pose, camera exposure, camera zoom, pedestal pose, pedestalangular position, object pose, sequences thereof, and/or otherparameters. Sampling parameters can be determined automatically,manually, be predetermined (e.g., a predetermined pose and lightingsequence), be dynamically determined (e.g., based on the images thathave already been sampled, object complexity, object parameters, etc.),or otherwise determined.

However, the image can be otherwise configured.

S100 preferably includes: receiving an object within the measurementvolume, controlling the system according to a set of samplingparameters, and sampling one or more images of the measurement volumeand/or object. S100 can optionally include determining one or more depthimages using the depth sensor(s) of the system (e.g., as depicted inFIGS. 3 and 4).

The object can be received: from a user, from the robotic system (e.g.,wherein the robotic system randomly grasps an object and places itwithin the measurement volume), from a conveyor, and/or be otherwisereceived.

The set of sampling parameters can be determined based on: apredetermined set of sampling parameters, the object's materialproperties (e.g., slower rotation rate when the object is deformable;backlighting and frontlighting the object when the object is translucentor reflective; etc.), and/or otherwise determined. Examples of samplingparameters that can be controlled include: the sampling rate, thepedestal rotation rate, the imaging system actuation rate, which lightsets are on or off, the lighting parameters of each light set (e.g.,color, temperature, intensity, hue, etc.), and/or the object positionwithin the measurement volume, and/or any other suitable parameter.

The images are preferably sampled by the imaging system (e.g., while thesystem is being operated according to the sampling parameters), but canbe otherwise sampled. The images can include 2D images, 3Drepresentations (e.g., depth map, point cloud, etc.), and/or any othersuitable image. One or more images can be contemporaneously (e.g.,concurrently, simultaneously, captured with a slight delay, etc.)sampled by the same or different image sensor, but be sampled atdifferent times. When the images are sampled by different sensors, thesensors are preferably coincident (e.g., have the same or similar poserelative to the object; have less than a threshold pose differencebetween the sensors; etc.), but can alternatively have different poses.When multiple images are contemporaneously sampled, the images arepreferably pixel-aligned, but can alternatively be misaligned. Forexample, an RGB image can include a scene depth value for all or aportion of the RGB image pixels. In a first variant, the pixel-alignedimages are sampled by the same sensor (e.g., a CCD or CMOS sensorsamples both the RGB images and the information used to extract depthfrom the scene). In a second variant, the pixel-aligned images aresampled by different sensors, wherein the respective measurements arealigned using a calibration for the sensor pair. However, the images canbe otherwise determined.

In a first variation, S100 includes: rotating the pedestal to apredetermined position, controlling the light emitting elementsaccording to the lighting parameters, moving the camera on an arc withrespect to the center of the pedestal, sampling the images, andrepeating the process until the termination event is met.

In a first example, the light emitting elements are controlled tofront-light the object.

In a second example, the light emitting elements are controlled tosequentially front-light then back-light the object (and/or vice versa),wherein multiple images (e.g., including a first image wherein theobject is front-lit and a second image wherein the object is back-lit)are sampled at each object position. This can help with object parameterdetermination.

In a third example, the light emitting elements are controlled tobacklight the object, which can enable object silhouette determination.

In a second variation, S100 includes continuously rotating the pedestalwhile the camera samples a series of images. In this variation, theobject pose relative to the camera can be determined based on the objectitself, based on the calibration features, and/or otherwise determined.In this variation, the lighting parameters can be constant for eachrotation, vary throughout the rotation, and/or be otherwise configured.

However, S100 can be determined using a combination of the above, orotherwise performed.

5.2 Generating an Object Mesh Using the Set of Images S200.

Generating an object mesh using the set of images can function todetermine a representation for the object that can be used to model theobject in a virtual scene. The object mesh is preferably determinedbased on the set of images, but can additionally or alternatively bedetermined based on any other suitable information. The object mesh caninclude a full object mesh for the object, multiple refined meshes thatrepresent different sides of the object (e.g., when the refined meshescannot be aligned), and/or any other suitable mesh.

Generating the object mesh (e.g., as shown in FIG. 12) can include:optionally determining a set of masks based on the set of images S220;determining a first mesh based on the set of images and/or the set ofmasks S240; optionally determining a refined mesh by recomputing thefirst mesh S260; optionally generating a full object mesh based on therefined mesh S280; and/or any other suitable elements.

Determining a set of masks based on the set images S220 can function toisolate the image segments depicting the object from the remainingcomponents of the system (e.g., pedestal, cavity, calibrationpattern(s), etc.) in the images (e.g., as depicted in FIG. 4 and FIG.5). The masks can be used to: generate a rough mesh (e.g., to determinewhether sufficient object data has been captured, such that a refinedmesh can be generated), used as a starting point for the improved masks,and/or otherwise used.

The masks are preferably indicative of the object pixels and/or area ineach image, but can alternatively be indicative of other systemcomponents appearing within the image. The masks are preferably highlyaccurate (e.g., include less than a threshold number of non-objectpixels, exclude less than a threshold number of object pixels, etc.),but can alternatively be inaccurate (e.g., include more than a thresholdnumber of non-object pixels, exclude more than a threshold number ofobject pixels), be a bounding box, and/or be any other mask. One or moremasks can be determined for each image. When the set of images includesfront lit images and back lit images, the back lit images are preferablyused to determine the object masks (e.g., as shown in FIG. 11). However,the front lit images can additionally or alternatively be used todetermine the object masks.

The masks are preferably determined after S100, but can alternatively bedetermined after S300 (e.g., using the object mesh) or otherwisedetermined. The masks are preferably initialized using the depth map(e.g., associated with the images), but can additionally oralternatively be initialized from the images themselves, from otherimages, and/or otherwise initialized. The masks can be refined using thefront-lit and/or back lit images, and/or refined using any otherinformation.

The masks can be determined using: image segmentation (e.g., usingclassical approaches, AI based techniques, etc.), depth map segmentation(e.g., thresholding the depth map at the pedestal height), meshreprojection, a combination thereof, and/or any other suitable method.

In a first variant, determining the set of masks can include:determining a point cloud from a depth map (e.g., using depthintegration); generating a rough mesh using the point cloud; andoptionally removing the pedestal from the rough mesh (e.g., using planeidentification and/or plane fitting, matching of a known pedestal planeto a plane in the point cloud, matching a known pedestal geometry to theestimated pedestal location within the point cloud, etc.). Aftergenerating the rough mesh, S220 can include: generating an initial maskby projecting the rough mesh into the one or more images, wherein theinitial mask includes binary values at each pixel location of the imagethat indicate if the pixel is part of the object; optionally croppingthe image to the mask; and refining the initial mask, such as by using agrabcut algorithm (e.g., wherein the masked segment is treated as thebounding box), a segmentation algorithm (e.g., neural network,clustering algorithm, etc.), and/or other image-based algorithms (e.g.,for color images, grayscale images, etc.). Refining the initial mask caninclude using one or more of the above algorithms to include or excludea pixel of the initial mask (e.g., relabel the pixel) based on whetherthe pixel is identified as an object pixel. The output of refining theinitial mask is preferably a masked image segment, but can be any othersuitable information.

In a second variant, determining the set of masks can includedetermining an object silhouette from back lit (and/or front litimages), and using the silhouette as a mask for each image. Determiningthe object silhouette can include determining background pixels based onintensity (e.g., having an expected intensity determined from thesampling lighting parameters; having a higher intensity in the back litimages; etc.), and segmenting the object pixels from the backgroundpixels.

However, the set of masks can be determined using a combination of theabove (e.g., in series, to provide votes for each pixel, etc.), orotherwise determined.

5.3 Determining a First Mesh Based on the Set of Masks S240.

Determining a first mesh based on the set of masks S240 can function todetermine a virtual object (e.g., volumetric object mesh, 3Dreconstruction, etc.) for use in scene modelling.

Parameters of the virtual object are preferably similar to and/or thesame as parameters of the physical object (e.g., be an accurate virtualmodel of the physical object), but can alternatively be different.Virtual object parameters can include: dimensions, geometry (e.g.,scaled, unsealed), visual parameters (e.g., object colors, colorlocations, surface parameters, etc.), and/or other parameters.

S240 is preferably performed after S220 (e.g., after determining the setof masks), but can additionally or alternatively be performed afterS100, and/or at any other suitable time.

The first mesh can be determined based on the set of images (e.g., fromS100, depth images (e.g., from S100), the rough mesh (e.g., from S220),the masked image segments (e.g., from S220), the set of masks (e.g.,from S220), and/or any other suitable data.

The first mesh can be determined using: monocular cues methods,stereovision methods (e.g., passive stereo, such as estimating depthfrom two or more RGB images by matching features across the images;active stereo, such as shining an IR pattern on the scene to generate apattern to be matched across two or more images; etc.), visual hullcarving, depth fusion, numerical approximation, a combination thereof,and/or any other suitable mesh generation method. Alternatively, thefirst mesh can be retrieved from a database (e.g., provided by an objectmanufacturer or distributor, from object CAD, etc.), or otherwisedetermined.

S240 preferably includes (e.g., as shown in FIG. 13): segmenting themeasurement to determine a volumetric object segment S242 anddetermining a first mesh based on the volumetric object segment S244.However, the first mesh can be otherwise determined.

Segmenting a measurement volume S242 functions to determine a detailedvolumetric representation of the object and/or segment the measurementvolume into object and non-object voxels. The volumetric object segmentcan be a point cloud, a set of voxels, and/or otherwise represented. Thevolumetric object segment can be denser than the depth map, sparser thanthe depth map, or otherwise related to the depth map. The volumetricobject segment can be determined based on the depth map, images, themasks (e.g., determined in S220), a combination thereof, and/or anyother suitable data. The measurement or object volume can be representedas an octree and/or any other suitable representation. The masks candefine a shape of the object from which to determine a detailedvolumetric representation of the object.

In a first variant, segmenting the measurement volume can be performedusing a visual hull algorithm (e.g., using volumetric carving), whereinthe set of masks are used as the foreground masks for the visual hullalgorithm. The visual hull algorithm can output an updated point cloud(e.g., a point cloud of the object, a set of voxels labeled with“object” or “background,” etc.), and optionally output a visual hull(e.g., bounding geometry of the object), wherein the updated point cloudis used to determine the first mesh.

In a second variant, segmenting the measurement volume can be performedusing a depth fusion algorithm (e.g., using a signed distance function,such as a truncated signed distance function (TSDF), fast marchingmethod, fast sweeping method, level-set method, etc.) using a maskeddepth map. Depth fusion algorithms can be particularly useful to detectobject concavities, or otherwise used. The depth map (e.g., sampled inS100) can be masked using the masks from S220, the visual hull from thefirst variant, and/or other inputs.

In a third variant, segmenting the measurement volume can be performedusing a combination of volumetric object segmentation techniques,wherein each voxel or point is assigned to the object and/or to anon-object (e.g., background) based on the votes from differentvariants.

In a specific example, S240 includes segmenting the measurement volumeusing multiple methods (e.g., visual hull carving and TSDF), whereineach method provides a vote on a per-voxel basis on whether the voxel isan object voxel or non-object voxel. Each vote can be weighted (e.g.,based on the votes generation technique) or unweighted. Voxels with morethan a threshold number of votes (and/or more than a threshold score,calculated from the votes) can be considered object voxels, while voteswith less than a threshold number of votes or score can be consideredbackground voxels.

However, the measurement volume can be otherwise segmented.

Determining a first mesh based on the volumetric object segment S244functions to generate a virtual geometric representation of the physicalobject. The first mesh is preferably generated from the object voxels ofthe volumetric representation of the object, but can additionally oralternatively be determined from the depth map, images, or other data.The first mesh is preferably generated by fitting a mesh to theoutermost object voxels, but can be otherwise generated. The first meshis preferably finer (e.g., includes more nodes, shorter edges, etc.)than the rough mesh used in S220, but can alternatively have the samecoarseness or be coarser. The first mesh is preferably not generatedfrom the rough mesh, but can alternatively be generated using the roughmesh (e.g., as a starting mesh that is refined, as a validationreference, etc.).

Determining the first mesh can include meshing the first mesh (e.g.,connecting points of the point cloud into oriented triangles). Meshingthe first mesh can be performed using Poisson meshing, Delaunaytriangulation, Voronoi-based surface reconstruction, a minimum spanningtree technique, and/or any other suitable surface reconstructionalgorithm.

However, the first mesh can be otherwise determined.

Determining a refined mesh by recomputing the first mesh S260 canfunction to determine a more accurate mesh for the object (e.g., furtherrefine the collection of vertices, edges, surfaces, and/or surfacenormals that define the object). In variants, determining the refinedmesh can function to adjust the shape and/or appearance of the firstmesh to closely match the object.

The refined mesh is preferably determined using the first mesh receivedfrom S240, but can additionally or alternatively be performed based onany other suitable information. The refined mesh can encompass half theobject (e.g., one hemisphere of the object), the entirety of the object,and/or any other suitable portion of the object.

Determining the refined mesh can include: determining refined masks byre-projecting the first mesh into the set of images; updating vertexlocations such that the rendered mesh is consistent with captured imagedata; refining the surface normals, such as using an algorithm (e.g.,BRDF, averaging, outlier exclusion, etc.); and/or any other suitableelement. Determining the refined mesh can be performed by: minimizingoffset between refined masks and the masks computed in S200; bymaximizing photoconsistency of the rendered mesh to the RGB imagescaptured in S100 (e.g., using gradient descent, triangulation, etc.);and/or by any other suitable optimization. The optimization canoptionally be alternated with smoothing (e.g. bilateral filtering,poison remeshing, etc.), such that the mesh is locally smooth but highlevel features such as edges and corners are preserved. Alternatively,the smoothing operation can be performed after the optimization, before,and/or at any other suitable time.

Determining the refined mesh can include: finding point correspondencesbetween the set of masks (e.g., from S220) and the refined masks or mesh(e.g., from S240) and triangulating vertex errors using the pointcorrespondences such that the refined masks are closer to the maskscomputed in S220. The point correspondences can be determined: using aclosest point algorithm; by detecting features and determiningcorrespondences based on the detected features, and/or using any othersuitable technique.

In a first variant, the point correspondences can be determined by:projecting the refined mesh into one or more camera views where therefined mesh corresponds to the object's silhouette in that camera view;and moving and/or adjusting (e.g., deforming) the refined mesh (e.g., in3D) so that the refined mesh matches the silhouette in all camera viewsusing a fine-grained mismatch technique that ingests the observedsilhouette as input.

In a second variant, the point correspondences can be determined bymatching a visual patch and/or feature of the refined mesh acrossmultiple camera views; and adjusting (e.g., deforming) the correspondingmesh element of the refined mesh (e.g., in 3D) to consistently generatea texture of the visual patch and/or feature in all (or a subsetthereof) the camera views.

Additionally or alternatively, the refined mesh can be determined usingadaptive mesh refinement, Delaunay refinement (e.g., Chew's secondalgorithm, Ruppert's algorithm, etc.), and/or any other suitablealgorithm.

However, the refined mesh can be otherwise determined.

Generating a full object mesh by combining multiple refined meshes S280can function to determine a 360-degree object mesh (e.g., combine meshesto form a complete 3D representation of the object). The full objectmesh is preferably generated using two or more refined meshes (e.g.,depicting different sides of the same object), but can alternatively begenerated from a single mesh, from the first mesh, from the rough mesh,and/or otherwise determined. The full object mesh can be: uncolorized,untextured, only include the object geometry, colorized, textured,and/or have any other suitable visual properties.

Generating the full object mesh can be performed before and/or afterS260, after S220, before and/or after S240, before and/or after S400,after multiple refined meshes have been determined for different sidesof the same object, after keypoints are added to each of the multiplerefined meshes, and/or performed at any other suitable time.

S280 can be performed based on one or more: image sets (e.g., obtainedfrom one or more instances of S100 for different views of the sameobject), the refined mesh(es) (e.g., from S260), and/or any othersuitable information. S280 can be performed by merging one or more imagesets, by retrieving material properties from a predetermined database(e.g., based on an object identifier), and/or otherwise determined.

In a first variant of the input to S280, a portion of the method (e.g.,S100 and S220-S260) can be performed to determine a first refined meshfor a first side of an object, the object can be flipped (e.g.,automatically using a robotic arm, manually, etc.), and be repeated todetermine a second refined mesh for a second side of the object. Thefirst and second refined meshes can be combined in S280.

In a second variant of the input to S280, two sets of images can bedetermined in S100, wherein the first set includes images of a firstside of the object and the second set includes images of a second sideof the object. The sets of images can be processed by S220-S260 inparallel and/or in series to determine a first and second refined mesh,which can be combined in S280.

In a first variant of S280, combining multiple refined meshes can beperformed by aligning refined meshes using features extracted from theassociated point clouds (e.g., depth maps), images, the refined meshes,and/or other features. In one example, the alignment features caninclude an extracted SIFT feature point cloud from each refined mesh,which can include matching keypoints across point clouds.

In a first embodiment, if a predetermined number of keypoints arematched, then the refined meshes are combined; otherwise the meshes arenot combined.

In a second embodiment, each SIFT keypoint (e.g., 3D, 2D, etc.) isassociated with one or more votes (e.g., number of images that includethe keypoint). When aligning multiple point clouds, there are somenumber of scene-layers associated with the keypoints that agree with thetransformation (e.g., determined from aligning keypoints). Eachscene-layer has a number of images that voted on it. There is apredetermined threshold on the number of votes per scene-layer and whenthe number of votes is above the threshold, the refined meshes arecombined.

In a second variant of S280, combining multiple refined meshes caninclude: initializing a transformation by flipping a refined meshupside-down and rotating the refined mesh along the z-axis in discretesteps; aligning the refined meshes after each rotation (e.g., for Nrotations) using a geometric alignment algorithm (e.g., ICP); andselecting the alignment with the minimum alignment error and/or based ona predetermined alignment error threshold. If the alignment error isabove the threshold, the refined meshes are not combined. The fullobject mesh can additionally or alternatively be otherwise determined.

However, the object mesh can be otherwise determined.

5.3 Determining One or More Object Components of the Object Mesh S300.

Determining the object components (e.g., faces, parts, etc.) of theobject mesh can function to determine and/or label connected componentsfor grasping, determine and/or label components for keypoint placementin S400, and/or any other suitable functionality. One or more objectcomponents can be determined for an object (e.g., 2, 4, 8, 11, etc.);however, the object can have a predetermined number of components (e.g.,6 faces, from a bounding box), or have any other suitable number ofcomponents. An object component is preferably an object face, but canadditionally or alternatively be an object part (e.g., handle, limb,head, body, lid, etc.), and/or any other component of an object. A faceis preferably on the mesh, but can additionally or alternatively be aface of a bounding box for the object (e.g., wherein points on thebounding box can be mapped to corresponding points on the object mesh)or any other suitable face. Each object component can be assigned aunique identifier (e.g., locally unique identifier for the object,globally unique identifier), or be unidentified. Each triangle of themesh can be assigned to component of the object that the mesh cell(e.g., triangle) is located on or cooperatively defines (and/or beassociated with the component's identifier), or be otherwise associatedwith a location on the object. Each vertex (e.g., of the triangle, ofthe mesh) can be assigned to the component that the vertex is locatedon, cooperatively defines, that the respective mesh cell is associatedwith (and/or be associated with the component's identifier), and/orotherwise assigned. Each component can be assigned one or more geometricobject properties which can be used to determine an object grasp.

Each object component can include one or more keypoints arranged alongthe component center, randomly distributed across the surface or volumeof the object component, arranged along the component boundaries,arranged in graspable regions, such as flat regions, equally distributedalong the object component surface or volume, be roughly geodesicallyequidistant on an object mesh (e.g., component tag mesh, object facesubmesh, etc.), and/or otherwise arranged.

In a first variant, the object components can be determined bysubdividing the object mesh (e.g., from S200) into connected components.The object mesh can be subdivided into object components based on localfeatures of the object mesh. The local features of the mesh geometry ofthe object mesh can include: geometric object properties (e.g., surfacecurvature, surface area, surface normals, etc.), planes, and/or anyother suitable features. The object components can be determined usingthe local features (e.g., using geometric object properties, planefitting, etc.); manually; by identifying contiguous surfaces, usingrules or heuristics (e.g., using contiguous surfaces betweendiscontinuities, based on surface normals, based on surface areas,etc.), and/or otherwise determined. In one example, different objectcomponents can be determined based on mesh normal continuity (e.g.,wherein different faces are identified when the mesh or surface normalchanges more than a threshold amount between adjacent mesh cells), agraspability function, and/or otherwise determined.

In an embodiment of the first variant, determining one or more objectcomponents can include computing vertex curvature; classifying verticesas curved if the curvature is above a threshold; computing a new mesh(e.g., component tag mesh), wherein the edges belonging to curvedvertices are removed; extracting connected components (submeshes orsubgraphs); and labelling each vertex according to a connected componentidentifier associated with the connected components. Computing vertexcurvature can include: collecting neighbor vertices within apredetermined radius distance (e.g., measured as the sum of edgelengths); representing the plane that the vertices lie on (e.g., usingthe mean vertex position and mean normal); measuring the distance ofeach vertex to the plane; and setting curvature to be the N-thpercentile of distances to the plane. However, the vertex curvature canbe otherwise computed.

In a second variant, the object components can be determined bygenerating a component tag mesh. The component tag mesh can function tosegment the mesh surface into component parts for labelling keypoints.The component tag mesh can be generated by building the component tagmesh from the triangles of the object mesh. Generating a component tagmesh can function to determine connected components of the object. Thecomponent tag mesh can be used for grasping the object (e.g.,determining a set of object grasp locations) and/or used for any othersuitable task. Vertices, mesh cells, and/or keypoints (of the face tagmesh) lying each object face can be labelled with the respective objectface's face tag, which can be used to retrieve the respective objectface's properties (e.g., material properties, geometric properties,etc.).

In a third variant, the object components can be determined based on abounding box surrounding the object. The bounding box can be determinedbased on the object mesh, such as fitting a rectangular box around theobject mesh such that the entire object mesh fits within the box and/orusing any other suitable technique. Determining the object componentsusing the bounding box can include: sampling points on a bounding boxgrid around the refined mesh; assigning a object component identifier tothose points based on which face of the bounding box they lie; mappingeach point to the nearest vertex on the object mesh; labelling thevertex on the object mesh with an object component identifier for thebounding box face from which the point was sampled (or otherwiselabelling the object mesh vertex); and/or any other suitable elements.The faces of the bounding box can be mapped to the same object componentbased on the object's shape (and labelled as the same object component)(e.g., for tubes, all faces but the one on the tube cap can be labelledas a single face; for cylinders, all faces but the two flat sides of thesurfaces can be labelled as a single face; etc.) and/or otherwiselabelled. Additionally or alternatively, surface normals can be used todetermine faces of the object (e.g., flat regions of tubes, cylinders,etc.). In a first example, a vertex (or triangle) can be assigned aobject component identifier if its vertices are close to a correspondingbounding box face and the normals are pointed in mostly the direction ofthe bounding box face.

However, the one or more object components can be otherwise determined.

5.4 Adding One or More Keypoints to the Refined Mesh S400.

Adding one or more keypoints to the refined mesh S400 can function tolabel the refined mesh with keypoints (e.g., per object component, perobject, etc.), wherein the keypoints can be indicative of possible grasppoints on the object and/or features of the object. The keypoints canlie on the object mesh, lie on a bounding box surrounding the objectmesh (e.g., be a minimum bounding box surrounding the object mesh),and/or be otherwise related to the object mesh.

The keypoints can be points of interest, points that can be matchedacross multiple images, keypoints indicative of the object bounding box(e.g., corner keypoints, face centroid keypoints, edge keypoints, etc.;example shown in FIG. 8B), object component keypoints (e.g., lying on aface of the object mesh, such as shown in FIG. 8A; lying on a vertex ofa face or part of the object; etc.), and/or any other suitable point.The keypoints can be 3D keypoints (e.g., identify a position and/or poserelative to the object), geometric keypoints, 2D keypoints, 2.5Dkeypoints (e.g., keypoint positioned relative to a given objectcomponent), and/or other keypoints. The keypoints can be independent ofvisual features (e.g., not determined based on visual cues, not entirelydetermined based on visual cues, etc.), be dependent on visual features(e.g., be traditional computer vision keypoints, such as locally uniquefeatures, edges, corners, etc.), and/or be otherwise defined. Invariants, the keypoints can be 3D keypoints (e.g., tied to the objectgeometry), but be identified in a 2D or 2.5D reference frame (e.g., theimage). The keypoints detected within the images (e.g., training imagesor inference images) can be for visible and/or hidden portions of theobject in the image.

Each keypoint can be associated with a keypoint identifier, anorientation (e.g., a surface normal of the object face, a surface normalof a bounding box face, etc.), a 3D position (e.g., in an object frameof reference), a position or pose relative to the object mesh (e.g.,fixed position on the object mesh, fixed position relative to the objectmesh, variable position relative to the object mesh, etc.), and/or anyother suitable information. The keypoint identifier can be: a boundingbox face identifier, a corner identifier, an object component identifier(face tag, such as determined from the face labels and/or face tag meshfrom S300) used to identify an object component, and/or any othersuitable identifier. The keypoint identifier can be an integer, label,or any other suitable value.

The keypoints can be added to the refined mesh per object component, perbounding box face, and/or otherwise added to the refined mesh. Thenumber of keypoints added per object component can be determined basedon the size of the component, can be a predetermined number (e.g., lessthan 5, more than 5, more than 10, more than 20, more than 50, more than100, a number that maximizes coverage of the object component, etc.),and/or the number of keypoints can be otherwise determined.

The keypoints can be evenly distributed across the object component,randomly distributed across the object component, added based on objectsurface normals (e.g., that are used to define the different objectcomponents, added to surfaces sharing similar surface normals, etc.),and/or otherwise placed on the refined mesh.

The keypoints can additionally or alternatively be ranked or prioritizedfor graspability (e.g., assigned a graspability score). The ranking canbe determined based on the respective surface normals, the respectiveobject face's area, the respective object face's texture, and/orotherwise determined.

In a first variant, the keypoints are bounding box keypoints, whereinthe keypoints are determined by sampling keypoints on the bounding box(e.g., faces, vertices, edges, etc.); and mapping sampled points from abounding box to a closest vertex on the object mesh. The keypoints addedto the object mesh in this way can be associated with the objectcomponent identifier corresponding to the side of the bounding box thatthe keypoint was sampled from. Alternatively, the bounding box keypointscan inherit the object component identifier of the closest objectcomponent.

In a second variant, the keypoints are local feature keypoints, whereinthe keypoints are assigned to object components that were determinedfrom the local features determined in S300.

However, the adding one or more keypoints can be otherwise performed.

5.5 Determining a Material Property Set for the Object S500.

Determining a material property set for the object S500 can function toenable more accurate object rendering in S700. The material property setcan be determined contemporaneously with S200-S400, performed afterS200-S400, and/or performed at any other suitable time. Materialproperties can include: shininess, reflectivity, transparency, color,texture, and/or any other suitable property. Each material property canbe associated with a score (e.g., value between 0-10, 0-100, −1 to 1,etc.), material type label, and/or otherwise characterized. The scorecan be determined using BRDF, and/or using any other suitable technique.The score can be determined based on a light measurement (intensity)from S100, based on brightness (e.g., how dark each pixel is), color,and/or any other information.

S500 can be performed based on: one or more image sets (e.g., obtainedfrom one or more instances of S100 for different views of the sameobject), the object mesh (e.g., from S200), and/or any other suitableinformation.

In a first variation, S500 includes determining object transparency.Object transparency can be determined based on: optical backgrounddistortions due to refraction, changes in chromaticity and brightnessdue to absorption, and/or using any other suitable method. In a firstexample, object transparency for all or portions of the object can bedetermined by evaluating the proportion or intensity of lighttransmitted through the object in a back lit image (e.g., as compared tothe background's light intensity and/or the amount of light emittedduring image capture), where the object pixels in the image can bedetermined based on the mask determined in S300 and/or refined meshdetermined in S400 (e.g., wherein the refined mesh is reprojected backinto the back lit image). However, object transparency can be otherwisedetermined.

In a second variation, S500 includes determining object specularityand/or reflectiveness. This variation can be used when the object isdetermined to be opaque (e.g., less than a threshold proportion of lightis transmitted through the object in the back lit image, as determinedusing the first variant), always used, or applied at any other suitabletime. The object specularity can be determined from the RGB images(e.g., from the same camera pose relative to the object with differentlighting patterns), and/or other data. The object specularity can bedetermined using a bidirectional reflectance distribution function(BRDF), such as the Lambertian model, Torrance-Sparrow model,Blinn-Phong model, and/or any other suitable BRDF model, but canadditionally or alternatively be determined using BSDF (bidirectionalscattering distribution function), Schlick's approximation, and/or anyother suitable method.

However, the material property set can be determined using a combinationof the above, or otherwise determined.

5.6 Modifying the Object Mesh S600.

Modifying the object mesh Shoo can function to modify the appearance ofthe object mesh by texturizing and/or coloring the object mesh (fromS200). The modified object mesh can be determined based on the set ofimages (e.g., the front lit images, front lit and back lit images,etc.), the material property set, and/or any other visual information.The material property set can be used to modify the color and/or textureof the object mesh determined from the set of images (e.g., weightingcolor and/or texture based on a property score, coloring and/ortexturing using the property score, etc.) and/or otherwise used tomodify the object mesh.

In a first variation, determining a texturized object mesh can includeadding color to the mesh. Adding color to the mesh can includeprojecting each vertex into the image and determining the color valuefrom the corresponding pixel. Alternatively, when multiple images areused, the color value can be determined by averaging the color valuesacross multiple images of the set of images, selecting the minimum,maximum, or otherwise selecting the color value for the vertex.

In a second variation, determining a texturized object mesh can includeadding texture to the mesh. In a first embodiment, adding texture to themesh can include projecting each triangle (and the vertices) into eachimage from S100 to determine based on a set of criteria (e.g., anglebetween triangle normal and direction to the camera, mean illumination,etc.) which image and associated pixel(s) to texture the triangle. Theresult of projecting each triangle into the mesh is a single textureimage from croppings of images from S100, wherein each triangle isassociated with a pixel coordinate identifier in the texture image foreach of the triangle's vertices.

However, the object mesh can be otherwise modified.

5.7 Determining Training Data S700.

Determining training data S700 can function to generate training datafor training the one or more object detectors in S800. The training datais preferably one or more synthetic or augmented images, but canadditionally or alternatively be one or more masks (e.g., binary masks),heatmaps, and/or any other suitable information. Each training dataelement (e.g., image, mask, heatmap, etc.) of the training data can beassociated with a training label (e.g., per training data element, perobject depicted in the training data element, etc.) (e.g., an example isdepicted in FIGS. 10A and 10B). The label can include: one or morekeypoints, object component identifiers (e.g., for each keypoint, foreach pixel, etc.), object component orientations (e.g., object faceorientation, object part orientation, etc.; for each keypoint, for eachpixel, etc.), visibility scores, occlusion scores, and/or any othersuitable information. In one example, the training label (e.g., trainingtarget) can include: an object keypoint associated with the portion ofthe object (mesh) depicted in the image, an object component label forthe object keypoint (and/or locally unique object keypoint identifier),and optionally a surface normal for the keypoint. However, the traininglabel can be otherwise constructed. The label can be a binary mask,vectors, values, and/or any other label. The label is preferablyunoccluded (e.g., keypoints that are in the field of view of a virtualcamera and not occluded by other object instances in the scene), but canadditionally or alternatively be occluded (e.g., keypoints not in thefield of view of a virtual camera and occluded by other object instancesin the scene, or on a surface of the object instance that is not in theview of the virtual camera).

The training data can be determined based on: one or more scenes (e.g.,rendering one or more scenes using a virtual camera's field of view);background images from various open source datasets (e.g., COCO, Boxbot,VOC, etc.) (e.g., rendering one or more object instances over thebackground image), and/or based on any other suitable information.

In some embodiments, generating the training data can includedetermining one or more scenes. Generating the training data canfunction to determine a scene (e.g., 2D, 3D, etc.) from which to capturetraining data (e.g., training images, such as using a virtual camera).The one or more scenes can be determined based on the object mesh (e.g.,full object mesh, multiple refined meshes of the same object, etc.),scene environment geometries, and/or any other information. The one ormore scenes can be determined using meshes from the same object or usingmeshes from multiple different objects. The one or more scenes can besimulated using the physics module, and/or any other simulation engine.The generated scene can include: the scene geometry, the scene's visualparameters (e.g., color, texture, etc.), and/or be otherwiseconstructed. S700 is preferably repeated multiple times to generatemultiple scenes (e.g., a predetermined number of scenes; iterativelydetermined until an object detector metric, such as accuracy, reaches apredetermined threshold; etc.), but can alternatively be performed once.

Determining the one or more scenes can include selecting (e.g.,randomly, quasi-randomly, pseudo-randomly, etc.): object types, objectnumerosity, poses (e.g., within the virtual grasping volume), lighting(e.g., color, position, etc.), fill level, object velocity, and/or anyother parameter. The scene can include: objects arranged in a pile,objects piled within a bin, objects distributed on a conveyor belt,objects arranged on a shelving unit, and/or any other suitableconfiguration of objects (e.g., as shown in FIG. 9). S700 can optionallysimulate object kinematics (e.g., based on gravity, object mass, etc.).For example, determining one or more scenes can include simulating theobjects falling into a bin, wherein the object pose within the bin canbe determined based on the falling kinematics.

In variants, when a scene is simulated using refined meshes that are notcombined (e.g., not combined in S200), the gravity vectors of each meshcan be tracked during simulation. When a gravity vector of a mesh facesa camera, the mesh can be flipped (e.g., such that a known object viewwill be rendered in S700).

In an embodiment of the above variants, simulating the scene can beperformed using two different refined meshes (e.g., when the meshescannot be combined into a full object mesh, such as when the object isdeformable). Simulating the scene can include simulating a first side ofa first object instance using a first refined mesh, and simulating afirst side of a second object instance using a second refined mesh,wherein the first refined mesh and the second refined mesh represent twodifferent sides of the same object. However, the scene can be otherwisesimulated using multiple refined meshes.

Determining one or more scenes can additionally include tracking theobject keypoints (from S400) for each object within the simulated scene(e.g., as depicted in FIG. 9). This can include: tracking whichkeypoints are visible (e.g., within a virtual camera's field of view),tracking the surface normals associated with the keypoints, and/ortracking any other suitable keypoint parameter.

Determining one or more scenes can optionally include calculating anocclusion or visibility score for each keypoint, object face (e.g., foreach object), object mask, and/or for a subset thereof. The occlusionscore can be used to train the object detector to determine an occlusionscore for each detected keypoint, or be otherwise used. The occlusion orvisibility score can be calculated by identifying the visible objectmesh cells that are closest to the virtual camera (across the entireobject population), and, for each face of each object, calculating theproportion of object face cells (e.g., mesh cells for the given objectcomponent or face) that were identified as visible object mesh cells.Alternatively, the calculating the occlusion score can include:determining mesh cells or keypoints closest to the camera (e.g., withthe shallowest depth or shortest distance to the synthetic camera) foreach object; determining whether there is an intervening object, meshcell, or keypoint between the object's closest mesh cells and thecamera; and scoring the keypoints without an intervening element with afirst score (e.g., “unoccluded”, “1”), and scoring the keypoints with anintervening element with a second score (e.g., “occluded”, “0”).Alternatively, calculating the occlusion score can include: projectingeach object's components into the synthetic camera's frame; scoring theelements corresponding to portions of the object appearing in thesynthetic image with a first score; and scoring the remaining elementswith a second score. However, the occlusion score can be otherwisedetermined and used.

However, the one or more scenes for determining training data can beotherwise determined or characterized.

Determining the training data can include rendering one or moresynthetic images of the one or more scenes based on one or more virtualcamera views (e.g., determined randomly, predetermined, etc.). The oneor more virtual camera views can be: centered above the scene (e.g.,with a top-down view), above and to the right or left of the scene,and/or any other suitable view. Rendering the synthetic image of thescene can include rendering projective shadows and/or shading (e.g.,determined based on the virtual camera view). Rendering the syntheticimage can be performed rendering different instances of the refined meshand/or full object mesh in the scene and using the refined surfacenormals.

Rendering the synthetic image of the scene can be performed using thematerial property set (e.g., to texture the synthetic images), using oneor more images from the set of images (e.g., preferably the front litimages, such as shown in FIG. 1i , but can additionally or alternativelyinclude the back lit images), and/or using any other information.Rendering the scene can be performed using OpenGL and/or any othersuitable API. Rendering can be performed based on the one or moreselected parameters from S700 and/or any other parameter values.

In a first variant (e.g., on-the-fly rendering variant), rendering asynthetic image of the scene can include performing, for each object inthe scene (or a subset thereof, such as the topmost objects or objectsclosest to the camera): determining the object pose of each object(and/or the topmost objects) relative to the virtual camera; selectingan image (e.g., preferably a front lit image, but can additionally oralternatively be a back lit image, ambiently-lit image, or other image)that is closest to the object pose (e.g., depicts the object in thespecified object pose relative to the camera) and optionally, furtherselecting the image based on image brightness (e.g., selecting an imagewith median brightness), lighting angle, and/or other visual parameters;projecting the image onto the mesh, projecting an image patch pertriangle onto the mesh, projecting the mesh into the image, and/orotherwise determining the visual elements associated with the visible orunobstructed object mesh cells; and rendering the masked or extractedimage segment from selected image as the rendered object image.

In a second variant (e.g., auto rendering variant), rendering asynthetic image of the scene can include performing, for each object inthe scene (or a subset thereof, such as the topmost objects or objectsclosest to the camera): determining an object pose of the object meshrelative to a virtual camera; determining an object texture for eachvisible triangle of the object mesh using all images of the set thatdepict the visible triangle (e.g., averaging the texture across all theimages); and projecting the object texture onto the mesh.

In a third variant, rendering a synthetic image of the scene can beperformed using image-based rendering, which can include approximatingan object pose transformation from a first image to a new pose using ahomography. Rendering a synthetic image of the scene in this way canrespect the 3D shape of the object or alternatively approximate theobject as a 2D plane. Rendering a synthetic image of the scene can beperformed using the material property set, front lit and/or back litimages, and/or any other suitable information.

In a fourth variant, rendering the synthetic image of the scene can beperformed by: determining colorized and/or textured versions of theobject mesh; generating the scene as discussed in S700; and rendering animage of the scene based on the portions of each object mesh in thescene that are visible to a synthetic camera (e.g., by projecting thecolored mesh cells into the camera frame). The colorized and/or texturedversions of the object mesh can be determined by projecting the objectmesh into the images sampled in S100, by retrieving the object mesh froma predetermined database, and/or otherwise determining the colorizedand/or textured object mesh.

After training data is determined from rendering the one or more scenes,the training data can be post-processed. Post-processing the trainingdata can include adjusting brightness, hue, saturation, and/or any otherparameter.

The training data can additionally include: one or more synthetic imagesof the one or more scenes (e.g., depicted in FIG. 9); one or morereal-world images of a real-world scene; masks for each object in thescene (e.g., determined by the method or real-world) and/or image (e.g.,modal mask, amodal mask, etc.); one or more keypoint locations in thesynthetic images (e.g., determined from projecting the keypoint from therespective object mesh into the virtual camera frame), associatedidentifiers (e.g., component tags, face tags, corner identifiers,bounding box face centroid identifiers, etc.), associated occlusionscores, associated graspability scores or rankings, and/or otherkeypoint data; the 3D bounding box for the object; object faceorientations; surface normals; face visibility scores; and/or any othersuitable information. The keypoints are preferably represented in the 2Dimage frame of reference (e.g., determined by projecting the 3D keypointlocation in the 3D scene into the camera's frame), but can be otherwiserepresented.

In a first variant, the keypoint locations in the synthetic images canbe determined using the scene simulation, which tracks the keypoints foreach object in the scene. Keypoint information that can be trackedinclude: which object instance that the keypoint belongs to, thekeypoint location relative to the object instance, the keypoint posewithin the scene, the keypoint tags or labels, and/or other information.After the scene is simulated, the unoccluded keypoints (e.g., closestkeypoint to the virtual camera within the virtual volume correspondingto a virtual camera's pixel) can be used as a training label for thesynthetic image. The keypoint locations can additionally oralternatively be determined (e.g., added to a training data label) basedon the orientation of a particular object mesh in the synthetic scene(e.g., using a pose of the object mesh to look up the unoccludedkeypoints for the object pose, and using the unoccluded keypoints as thetraining label). In a first embodiment of this variant, the unoccludedobject keypoints in the synthetic scene can be represented as a binarymask, wherein the binary mask is used to train the object detector toidentify unoccluded object keypoints in synthetic image.

In a second variant, the keypoint locations in the synthetic images canbe determined by tracking object components in the scene simulation, andsampling keypoints from the object components in the synthetic image asthe unoccluded keypoints for the training label. Additionally oralternatively, the object components can be used as the training label(e.g., unoccluded portions of the object components).

However, the training data can be otherwise determined.

5.8 Training One or More Object Detectors Using the Training Data S800.

Training one or more object detectors using the training data S800 canfunction to train one or more object detectors for object detection(e.g., detecting keypoints based on images). The one or more objectdetectors can be trained for the same object, different objects (e.g.,each detector is trained to detect keypoints for a particular object),and/or the one or more detectors can be otherwise trained. Training theone or more object detectors can be performed by the system, by aseparate system, and/or any other suitable system. The one or moredetectors can receive the synthetic image and/or training data image asinput. The one or more detectors can be trained to determine thetraining label associated with the synthetic image and/or training dataimage. The one or more object detectors can be a: neural network (e.g.,feed forward neural network recurrent neural network, convolutionalneural network, etc.); Viola-Jones object detection model;Scale-invariant feature transform model (SIFT), Histogram of orientedgradients model (HOG), graphical model (e.g., RCN, Bayesian network,etc.), and/or any other suitable detector. The convolutional neuralnetwork can be a: you only look once (YOLO), region proposal network(e.g., R-CNN, cascade R-CNN, Fast R-CNN, Faster R-CNN), a single shotmultibox detector (SSD), a retina-net, deformable convolutionalnetworks, and/or any other suitable neural network. Training the one ormore object detectors can include using an optimization algorithm tofind a set of weights for the object detector that best maps the inputimage to detected keypoints. The optimization algorithm can be: gradientdescent, stochastic gradient descent, mini-batch gradient descent, Adamalgorithm, Nesterov accelerated gradient algorithm, Adagrad algorithm,AdaDelta algorithm, and/or any other suitable algorithm.

A first specific example of the network is described in U.S. applicationSer. No. 13/895,225 filed 16 May 2013, which is incorporated herein inits entirety by this reference.

A second specific example of the network is described in U.S.application Ser. No. 15/803,595 filed 3 Nov. 2017, which is incorporatedherein in its entirety by this reference.

In a first variant, training the one or more object detectors caninclude selecting the network to train based on the training data, basedon the object, and/or based on any other suitable information.

In a second variant, training the one or more detectors can includetraining an object specific neural network to classify whether an objectof interest is within an insertion target or not (e.g., a predetermineddistance from the insertion target, such as more than 1 mm, more than 5mm, more than 1 cm, more than 2 cm, etc.), such as for post-insertioninspection. In this variant, the input can be a synthetic image or areal world image of an object inserted into an insertion component andlabelled based on the position of the object relative to the insertioncomponent.

However, the one or more object detectors can be otherwise trained.

5.9 Detecting Objects Using the Trained Object Detector S900

Detecting objects using the trained object detector S900 can function todetect objects (e.g., object bounding boxes) and/or object features(e.g., keypoints, and optionally associated object componentidentifiers) in deployment environments based on sampled images of thescene (e.g., real-world scene). The object and/or feature detections canbe used to: select an object to grasp, select a contact (e.g., grasp)point, avoid the detected object or feature, and/or otherwise used.

Deployment environments can include: object grasping environments,object placing environments (e.g., keypoints of an insertion volume),navigation environments (e.g., for autonomous land vehicles; forautonomous water vehicles; for autonomous aircraft; for autonomousrobots, such as outdoor or indoor robots; etc.), indoor and/or outdoorscenes, and/or any other suitable deployment environment.

In a first variant, the deployment environment is a physical scene(e.g., a conveyor belt, a container, etc.) that includes one or moreobjects (e.g., overlapping objects; such as a pile of objects; nonoverlapping objects, etc.). In a first example, the deploymentenvironment is a pile of multiple different versions of the same object.In a second example, the deployment environment is a pile of multipledifferent versions of different objects. In a third variant, thedeployment environment is a shelving unit, wherein objects can begrasped on a particular shelf of the shelving unit.

In a second variant, the deployment environment is a physical navigationspace (e.g., road, side walk, side of the road, etc.).

The detected keypoints output by the trained network can be used toenable: object grasping, object insertion, image stitching by matchingthe keypoints across different images, navigation (e.g., objectavoidance and/or object presence to enable more accurate navigation),and/or any other suitable application. Optionally, the detectedkeypoints can be used to determine object presence in the image, objectlocation in the image, a graspable point on the detected object, and/orotherwise used. However, objects can be otherwise detected.

In a first variant, detecting objects using the trained object detectorcan include: capturing an image of a physical scene using an imagingsystem (e.g., preferably a different imaging system than that used tocapture the set of images in S100, but can additionally or alternativelybe the same imaging system) proximal the physical scene; and detectingan inference keypoint, associated with an object component identifier,of an object in the physical scene using the trained object detector.

In a second variant, detecting objects using the trained object detectorcan include capturing an image of a navigation environment using animaging system; and detecting an inference keypoint, associated with anobject component identifier, of an object in the navigation environmentusing the trained detector, and using the detected inference keypoint toenable more accurate navigation (e.g., object avoidance, steering,etc.). However, the object detector can be otherwise used to detectobjects.

Embodiments of the system and/or method can include every combinationand permutation of the various system components and the various methodprocesses, wherein one or more instances of the method and/or processesdescribed herein can be performed asynchronously (e.g., sequentially),concurrently (e.g., in parallel), or in any other suitable order byand/or using one or more instances of the systems, elements, and/orentities described herein.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the preferred embodiments of the invention withoutdeparting from the scope of this invention defined in the followingclaims.

We claim:
 1. A method, comprising: generating an object mesh comprisinga set of object components, wherein the object mesh is associated with aplurality of object keypoints, each labeled with an object componentidentifier for an object component of the set that the object keypointis proximal to; rendering a synthetic image using the object mesh;associating the synthetic image with object keypoints of the pluralitythat are unoccluded in the synthetic image; and training an objectdetector to identify the unoccluded object keypoints in the syntheticimage.
 2. The method of claim 1, wherein the object mesh is generatedfrom a set of images, wherein the set of images is captured using animage capture environment.
 3. The method of claim 2, wherein the set ofimages comprises depth images, wherein the depth images comprise surfacenormals for the points in each image, and determining refined surfacenormals using bidirectional reflectance, wherein determining thesynthetic image comprises rendering the object mesh using the refinedsurface normals.
 4. The method of claim 2, wherein capturing the set ofimages comprises: controlling a set of light emitting elements toback-light the object; capturing a first image at a camera position whenthe object is back-lit; controlling the set of light emitting elementsto front-light the object; and capturing a second image at the cameraposition when the object is front-lit; wherein the object mesh isgenerated using the first image, and the synthetic image is determinedusing the second image.
 5. The method of claim 2, wherein rendering thesynthetic image comprises: determining a virtual object pose relative tothe object mesh; selecting an image of the set of images associated witha sampling camera position closest to the virtual object pose; anddetermining visual features of the object based on the selected image.6. The method of claim 1, wherein the unoccluded object keypoints arerepresented as a binary mask, wherein the binary mask is used to trainthe object detector to identify unoccluded object keypoints in images.7. The method of claim 1, wherein the plurality of object keypoints thatare associated with a shared object component identifier are randomlydistributed across an object component identified by the objectcomponent identifier.
 8. The method of claim 1, wherein the keypointsare determined using a bounding box around the object.
 9. The method ofclaim 1, wherein the plurality of object keypoints are associated withthe object mesh based on local features of the object mesh.
 10. Themethod of claim 1, wherein the object mesh represents a first side of anobject, wherein the method further comprises generating a second objectmesh that represents a second side of the object opposing the firstside, wherein rendering a synthetic image comprises rendering a firstside of a first object instance using the object mesh and rendering asecond side of a second object instance using the second object mesh.11. The method of claim 1, wherein the keypoints represent graspablepoints on the object.
 12. The method of claim 1, further comprisingidentifying a grasp point based on keypoints detected by the trainedobject detector for an inference instance of the object; and graspingthe inference instance of the object at the grasp point using a roboticarm.
 13. A method, comprising: capturing an image of a physical sceneusing an imaging system proximal the physical scene; and detecting aninference keypoint, associated with an object component identifier foran object component of an object in the physical scene using an objectdetector trained on training data generated by: determining an objectmesh associated with a plurality of object keypoints that are eachassociated with a position relative to an object mesh of an objectgeometry and a respective object component identifier; rendering asynthetic image using the object mesh; and training the object detectorto detect the object keypoints of the object mesh within the syntheticimage.
 14. The method of claim 13, wherein the physical scene comprisesa pile of multiple different versions of the same object.
 15. The methodof claim 13, wherein the object keypoints associated with the objectmesh are used as a ground truth label for the synthetic image.
 16. Themethod of claim 13, wherein the object mesh is generated from a set ofimages that depict two different sides of the object, and whereingenerating the object mesh comprises combining meshes determined foreach side of the object.
 17. The method of claim 13, wherein the objectmesh is generated from a set of images that is captured using an imagecapture environment, and wherein the set of images cooperatively form aphoto dome around the object.
 18. The method of claim 13, wherein theobject keypoints are bounding box keypoints.
 19. The method of claim 16,wherein the plurality of object keypoints are associated with the objectmesh by sampling keypoints on a bounding box around the object mesh. 20.The method of claim 19, wherein the object component identifierassociated with a particular object keypoint of the plurality isassigned based on a face of the object mesh that the object keypoint islocated on.
 21. The method of claim 13, wherein a graspable point on theobject is determined based on the inference keypoint.
 22. The method ofclaim 13, wherein the plurality of object keypoints that are assigned toan object component are randomly distributed across the objectcomponent.